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A NEURAL NETWORK ASSISTED MULTI -SPECTRAL SEGMENTATION SYSTEM 
CROSS-REFERENCE TO RELATED APPLICATION 

[0001] This application is a continuation of U.S. Application 
Serial No. 09/040,378, filed March 18, 1998, the disclosure of 
which is hereby incorporated by reference herein. The *378 
application is the national phase of International Application 
No. PCT/CA 96/00619, filed September 18, 1996, the disclosure 
of which is hereby incorporated by reference herein. This 
application also claims benefit of U.S. Provisional 
Application No. 60/003,964, filed September 19, 1995, the 
disclosure of which is hereby incorporated by reference 
herein . 

FIELD OF THE INVENTION 

[0002] The present invention relates to automated diagnostic 
techniques in medicine and biology, and more particularly to 
multi-spectral segmentation of nuclear and cytoplasmic 
objects . 

BACKGROUND OF THE INVENTION 

[0003] Automated diagnostic systems in medicine and biology 
often rely on the visual inspection of microscopic images. 
Known systems attempt to mimic or imitate the procedures 
employed by humans. An appropriate example of this type of 
system is an automated instrument designed to assist a cyto- 
technologist in the review or diagnosis of Pap smears. In its 
usual operation such a system will rapidly acquire microscopic 
images of the cellular content of the Pap smears and then 
subject them to a battery of image analysis procedures. The 
goal of these procedures is the identification of images that 
are likely to contain unusual or potentially abnormal cervical 
cells . 

[0004] The image analysis techniques utilized by these 
automated instruments are similar to the procedures 
consciously, and often unconsciously, performed by the human 
cytotechnologist . There are three distinct operations that 
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must follow each other for this type of evaluation: (1) 
segmentation; (2) feature extraction; and (3) classification. 
10005] The segmentation is the delineation of the objects of 
interest within the micrographic image. In addition to the 
cervical cells required for an analysis there is a wide range 
of "background" material, debris and contamination that 
interferes with the identification of the cervical cells and 
therefore must be delineated. Also for each cervical cell, it 
is necessary to delineate the nucleus with the cytoplasm. 
[0006] The Feature Extraction operation is performed after the 
completion of the segmentation operation. Feature extraction 
comprises characterizing the segmented regions as a series of 
descriptors based on the morphological, textural, 
densitometric and colorimetric attributes of these regions. 
[0007] The Classification step is the final step in the image 
analysis. The features extracted in the previous stage are 
used in some type of discriminant-based classification 
procedure. The results of this classification are then 
translated into a "diagnosis" of the cells in the image. 
[0008] Of the three stages outlined above, segmentation is the 
most crucial and the most difficult. This is particularly true 
for the types of images typically encountered in medical or 
biological specimens . 

[0009] In the case of a Pap smear, the goal of segmentation is 
to accurately delineate the cervical cells and their nuclei. 
The situation is complicated not only by the variety of cells 
found in the smear, but also by the alterations in morphology 
produced by the sample preparation technique and by the 
quantity of debris associated with these specimens. 
Furthermore, during preparation it is difficult to control the 
way cervical cells are deposited on the surface of the slide 
which as a result leads to a large amount of cell overlap and 
distortion . 

[0010] Under these circumstances a segmentation operation is 
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difficult. One known way to improve the accuracy and speed of 
segmentation for these types of images involves exploiting the 
differential staining procedure associated with all Pap 
smears. According to the Papanicolaou protocol the nuclei are 
stained dark blue while the cytoplasm is stained anything from 
a blue-green to an orange -pink. The Papanicolaou Stain is a 
combination of several stains or dyes together with a specific 
protocol designed to emphasize and delineate cellular 
structures of importance for pathological analysis. The stains 
or dyes included in the Papanicolaou Stain are Haematoxylin, 
Orange G and Eosin Azure (a mixture of two acid dyes, Eosin Y 
and Light Green SF Yellowish, together with Bismark Brown) . 
Each stain component is sensitive to or binds selectively to a 
particular cell structure or material. Haematoxylin binds to 
the nuclear material coloring it dark blue. Orange G is an 
indicator of keratin protein content. Eosin Y stains nucleoli, 
red blood cells and mature squamous epithelial cells. Light 
Green SF yellowish acid stains metabolically active epithelial 
cells. Bismark Brown stains vegetable material and cellulose. 
[0011] The combination of these stains and their diagnostic 
interpretation has evolved into a stable medical protocol 
which predates the advent of computer-aided imaging 
instruments. Consequently, the dyes present a complex pattern 
of spectral properties to standard image analysis procedures. 
Specifically, a simple spectral decomposition based on the 
optical behavior of the dyes is not sufficient on its own to 
reliably distinguish the cellular components within an image. 
The overlap of the spectral response of the dyes is too large 
for this type of straight - forward segmentation. 
[0012] The use of differential staining characteristics is 
only the means to the end in the solution to the problem of 
segmentation. Of equal importance is the procedure for 
handling the information provided by the spectral character of 
the cellular objects when making a decision concerning 
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identity . 

[0013] In the art, attempts have been made to automate 

diagnostic procedures, however, there remains a need for a 
system for performing the segmentation process. 
BRIEF SUMMARY OF THE INVENTION 

[0013] The present invention provides a Neural -Network 
Assisted Multi-Spectral Segmentation (also referred to as the 
NNA-MSS) method and system. 

[0014] The first stage according to the present invention 
comprises the acquisition of three images of the same 
micrographic scene. Each image is obtained using a different 
narrow band-pass optical filter which has the effect of 
selecting a narrow band of optical wavelengths associated with 
distinguishing absorption peaks in the stain spectra. The 
choice of optical wavelength bands is guided by the degree of 
separation afforded by these peaks when used to distinguish 
the different types of cellular material on the slide surface. 

[0015] The second stage according to the invention comprises a 
neural -network (trained on an extensive set of typical 
examples) to make decisions on the identity of material 
already deemed to be cellular in origin. The neural network 
decides whether or not a picture element in the digitized 
image is nuclear or not nuclear in character. With the 
completion of this step the system can continue on applying a 
standard range of image processing techniques to refine the 
segmentation. The relationship between the cellular components 
and the transmission intensity of the light images in each of 
the three spectral bands is a complex and non- linear one. By 
using a neural network to combine the information from these 
three images it is possible to achieve a high degree of 
success in separating the cervical cell from the background 
and the nuclei from the cytoplasm. A success that would not be 
possible with a set of linear operations alone. 

[0016] The diagnosis and evaluation of Pap smears is aided by 
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the introduction of a differential staining procedure called 
the Papanicolaou Stain. The Papanicolaou Stain is a 
combination of several stains or dyes together with a specific 
protocol designed to emphasize and delineate cellular 
structures of importance to pathological analysis. The stains 
or dyes included in the Papanicolaou Stain are Haematoxy.lin, 
Orange G and Eosin Azure (a mixture of two acid dyes, Eosin Y 
and Light Green SF Yellowish, together with Bismarck Brown) . 
Each stain component is sensitive to or binds selectively to a 
particular cellular structure or material. Haematoxylin binds 
to the nuclear material coloring it dark blue; Orange G is an 
indicator of keratin protein content; Eosin Y stains nucleoli, 
red blood cells and mature squamous epithelial cells; Light 
Green SF yellowish stains metabolically active epithelial 
cells; Bismarck Brown stains vegetable material and cellulose, 

[0017] According to another aspect of the invention, three 
optical wavelength bands are used in a complex procedure to 
segment Papanicolaou- stained epithelial cells in digitized 
images. The procedure utilizes standard segmentation 
operations (erosion, dilation, etc.) together with the neural- 
network to identify the location of nuclear components in 
areas already determined to be cellular material. 

[0018] The purpose of the segmentation is to extract the 
cellular objects, i.e. to distinguish the nucleus of the cell 
from the cytoplasm. According to this segmentation the multi- 
spectral images are divided into two classes: cytoplasm 
objects and nuclear objects, which are separated by a multi- 
dimensional threshold t which comprises a 3 -dimensional space. 

[0038] The neural network according to the invention comprises 
a Probability Projection Neural Network (PPNN) . The PPNN 
according to the present invention features fast training for 
a large volume of data, processing of multi-modal non-Gaussian 
data distribution, good generalization simultaneously with 
high sensitivity to small clusters of patterns representing 
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the useful subclasses of cells. In another aspect, the PPNN is 
implemented as a hardware -encoded algorithm. 

[0019] A method of analyzing cells comprises providing a 
plurality of digitized images of at least one cell of regions 
of unknown nuclear or cytoplasmic material. Each digitized 
image is formed from an optical image having a plurality of 
pixels associated therewith. Each digitized image is formed 
in a narrow band of optical wavelength different from the 
other digitized images. Each of the plurality of pixels has 
values from each of the digitized images. The method further 
includes analyzing the values for each pixel to identify 
nuclear and cytoplasmic material of the at least one cell of 
unknown regions of nuclear or cytoplasmic material . The 
analyzing step utilizes previously developed classification 
information for discriminating nuclear or cytoplasmic 
material. The previously developed classification information 
is developed from at least one cell of known regions of 
nuclear or cytoplasmic material. 

[0020] The step of analyzing the values for each pixel 
preferably includes utilizing a classifier trained on a 
training set of data developed from images having known 
regions of nuclear and cytoplasmic material. 

[0021] In a preferred embodiment, the previously developed 
classification information preferably includes values for 
pixels stored in a look-up table in a memory storage device. 
The previously developed classification information preferably 
has memory addresses in the look-up table, the memory 
addresses comprising a concatenation of the values from each 
of the digitized images representing the same region of the at 
least one cell. Each of the digitized images are drawn from a 
band of optical wavelength. 

[0022] In a preferred embodiment, the previously developed 
classification information includes a predetermined 
discriminant between values for pixels representing regions of 
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nuclear and cytoplasmic material . 

[0023] The method preferably includes the step of assigning a 
classification to each pixel as representing nuclear or 
cytoplasmic material . 

[0024] In a preferred embodiment, an absorption map is formed 
from each of the digitized images to represent the light 
absorption characteristics associated with each value for each 
pixel before the step of analyzing the values for each pixel. 
A classification is assigned to each pixel based upon 
absorption characteristics. The step of forming an absorption 
map preferably comprises applying a formula to each of the 
digitized images. 

[0025] The step of analyzing may include applying a linear 
discriminant analysis to define a linear boundary between 
values for pixels representing regions of nuclear and 
cytoplasmic material . The linear discriminant analysis 

preferably discriminates between pixels of nuclear material 
and at least two types of cytoplasmic material . 

[0026] The at least one cell of unknown nuclear or cytoplasmic 
material may comprise, for example, a cellular sample prepared 
according to the Papanicolaou staining procedure. 

[0027] The previously developed classification information may 
be developed by: providing a plurality of digitized images of 
at least one cell of regions of known nuclear or cytoplasmic 
material, each digitized image being formed from an optical 
image having a plurality of pixels associated therewith, each 
digitized image being formed in a narrow band of optical 
wavelength different from the other digitized images, each of 
the plurality of pixels having values from each of the 
digitized images; assigning a classification to each pixel as 
representing regions of nuclear or cytoplasmic material; and 
storing the classifications in a look-up table in a memory 
storage device. 

[0028] The step of storing preferably includes storing the 
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classifications in locations of the look-up table, the 
locations having addresses corresponding to the pixels. 
In a preferred embodiment, the step of analyzing comprises 
neural network processing of the values for pixels of the 
digitized images. The neural network processing comprises 
accepting one or more inputs into at least one processing 
element, multiplying the inputs by weighing factors, and 
applying a formula to the weighed inputs to provide an output 
to a plurality of other processing elements. Each value for a 
pixel is preferably accepted and processed by a different one 
of the at least one processing elements. 

[0029] In another aspect of the present invention, a method of 
analyzing cells comprises providing a plurality of digitized 
images of at least one cell. Each digitized image is formed 
in a narrow band of optical wavelength different from the 
other digitized images and including: at least one first 
digitized image in a wavelength of between 525 to 575 
nanometers; at least one second digitized image in a 
wavelength of between 565 to 582 nanometers; and at least one 
third digitized image in a wavelength of between 625 to 635 
nanometers. The digitized images are analyzed to identify 
nuclear and cytoplasmic material of the at least one cell. 
[0030] For example, the at least one first digitized image may 
have a wavelength of between 52 5 to 53 5 nanometers, the at 
least one second digitized image may have a wavelength of 
between 572 to 582 nanometers, and the at least one third 
digitized image may have a wavelength of between 625 to 635 
nanometers . 

[0031] For example, the at least one first digitized image may 
have a wavelength of between 535 to 545 nanometers, the at 
least one second digitized image may have a wavelength of 
between 572 to 582 nanometers, and the at least one third 
digitized image may have a wavelength of between 625 to 635 
nanometers . 




8 



VERACE 3.0-010 CONT 

[0032] For example, the at least one first digitized image may 
have a wavelength of between 565 to 575 nanometers, the at 
least one second digitized image may have a wavelength of 
between 565 to 575 nanometers, and the at least one third 
digitized image may have a wavelength of between 625 to 635 
nanometers . 

[0033] The step of analyzing the pixels preferably includes 
utilizing a classifier trained on a set of data developed from 
images having known regions of nuclear and cytoplasmic 
material. The step of analyzing preferably includes analyzing 
the digitized images based upon previously developed 
classification information . 

[0034] The step of analyzing may comprise analyzing values for 
each pixel, each pixel having values from each of the 
digitized images. 

[0035] A classification is preferably assigned to each pixel 
as representing regions of nuclear or cytoplasmic material. 
The at least one cell may comprise, for example, a cellular 
sample prepared according to the Papanicolaou staining 
procedure . 

[0036] The previously developed classification information is 
preferably developed from at least one cell of known regions 
of nuclear or cytoplasmic material for analyzing cells of 
unknown regions of nuclear or cytoplasmic material. 

[0037] The method, in certain preferred embodiments, includes 
the step of storing the previously developed classification 
information in a look-up table in an electronic memory device. 
In certain preferred embodiments, the step of analyzing may 
comprise neural network processing of the digitized images. 

[0039] In a further aspect, the present invention provides a 
method for identifying nuclear and cytoplasmic objects in a 
biological specimen, said method comprising the steps of: (a) 
acquiring a plurality of images of said biological specimen; 

(b) identifying cellular material from said images and 
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creating a cellular material map; (c) applying a neural 
network to said cellular material map and classifying nuclear 
and cytoplasmic objects from said images. 

[0040] In another aspect, the present invention provides a 
system for identifying nuclear and cytoplasmic objects in 
biological specimen, said system comprising: (a) image 
acquisition means for acquiring a plurality of images of said 
a biological specimen; (b) processing means for processing 
said images and generating a cellular material map identifying 
cellular material; (c) neural processor means for processing 
said cellular material map and including means for classifying 
nuclear and cytoplasmic objects from said images. 
[0041] In another aspect, the present invention provides a 
hardware -encoded neural processor for classifying input data, 
said hardware -encoded processor comprising: (a) a memory 
having a plurality of addressable storage locations; (b) said 
addressable storage locations containing classification 
information associated with the input data; (c) address 
generation means for generating an address from said input 
data for accessing the classification information stored in 
said memory for selected input data. 

[0042] A preferred embodiment of the present invention will 
now be described, by way of example, with reference to the 
following specification, claims, and drawings. 
BRIEF DESCRIPTION OF THE DRAWINGS 

[0043] Fig. 1 shows in flow chart form a neural network 
assisted multi-spectral segmentation method according to the 
present invention; 

[0044] Fig. 2 shows in diagrammatic form a processing element 
for the neural network; 

[0045] Fig. 3 shows in diagrammatic form a neural network 
comprising the processing elements of Fig. 2; 

[0046] Fig. 4 shows in diagrammatic form a training step for 
the neural network; 
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[0047] Fig. 5 shows in flow chart form a clustering algorithm 
for the neural network according to the present invention; and 

[0048] Fig. 6 shows a hardware implementation for the neural 
network according to the present invention. 
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

[0049] The present invention provides a Neural Network 
Assisted Multi-Spectral Segmentation (also referred to as NNA- 
MSS) system and method. The multi- spectral segmentation method 
is related to that described and claimed in co-pending 
International Patent Application No. CA96/00477 filed 
July 18, 1996 and in the name of the applicant. 

[0050] The NNA-MSS according to the present invention is 
particularly suited to Papanicolaou-stained gynaecological 
smears and will be described in this context. It is however to 
be understood that the present invention . has wider 
applicability to applications outside of Papanicolaou-Stained 
smears . 

[0051] Reference is first made to Fig. 1 which shows in flow 
chart a Neural Network Assisted Multi-Spectral Segmentation 

(NNAMSS) method 1 according to the present invention. 

[0052] The first step 10 involves inputting three digitized 
images, i.e. micrographic scenes, of a cellular specimen. The 
images are taken in each of the three narrow optical bands: 
540 ± 5 nm; 577 ± 5 nm and 630 ± 5 nm, (The images are 
generated by an imaging system (not shown) as will be 
understood by one skilled in the art, and thus need not be 
described in detail here.) The images are next processed by 
the multi-segmentation method 1 and neural network as will be 
described. 

[0053] As shown in Fig. 1, the images are subjected to a 
leveling operation (block 12) . The leveling operation 12 
involves removing the spatial variations in the illumination 
intensity from the images. The leveling operation is 
implemented as a simple mathematical routine using known image 
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processing techniques. The result of the leveling operation is 
a set of 8-bit digitized images with uniform illumination 
across their fields. 

[0054] The 8-bit digitized images first undergo* a series of 
processing steps to identify cellular material in the 
digitized images. The digitized images are then processed by 
the neural network to segment the nuclear obj ects from the 
cytoplasm objects. 

[0055] Referring to Fig. 1, following the leveling operation 
12 the next operation comprises a threshold procedure block 
14 . The threshold procedure involves analyzing the leveled 
images in a search for material of cellular origin. The 
threshold procedure 14 is applied to the 53 0 nm and 63 0 nm 
optical wavelength bands and comprises identifying material in 
the image of cellular origin as regions of the digitized image 
that fall within a range of specific digital values. The 
threshold procedure 14 produces a single binary "map" of the 
image where the single binary bit identifies regions that are, 
or are not, cellular material. 

[0056] The threshold operation 14 is followed by a dilation 
operation (block 16) . The dilation operation 16 is a 
conventional image processing operation which modifies The 
binary map of cellular material generated in block 14 . The 
dilation operation allows the regions of cellular material to 
grow or dilate by one pixel in order to fill small voids in 
large regions. Preferably, the dilation operation 16 is 
modified with the condition that the dilation does not allow 
two separate regions of cellular material to join to make a 
single region, i.e. a "no- join" condition. This condition 
allows the accuracy of the binary map to be preserved through 
dilation operation 16. Preferably, the dilation operation is 
applied twice to ensure a proper filling of voids. The result 
of the dilation operations 16 is a modified binary map of 
cellular material. 
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[0057] As shown in Fig. 1, the dilation operation 16 is 
followed by an erosion operation (block 18) . The erosion 
operation 18 brings the modified binary map of cellular 
material (a result of the dilation operation 16) back to its 
original boundaries. The erosion operation 18 is implemented 
using conventional image processing techniques. The erosion 
operation 18 allows the cellular boundaries in the binary 
image to shrink or erode but will not affect the filled voids. 
Advantageously, the erosion operation 18 has the additional 
effect of eliminating small regions of cellular material that 
are not important to the later diagnostic analysis. The result 
of the erosion operation 18 is a final binary map of the 
regions in the digitized image that are cytoplasm. 

[0058] The next stage according to the invention, is the 
operation of the neural network at block 20. The neural 
network 20 is applied to the 8-bit digitized images, with 
attention restricted to those regions that lie within the 
cytoplasm as determined by the final binary cytoplasm map 
generated as a result of the previous operations. The neural 
network 2 0 makes decisions concerning the identity of 
individual picture elements (or "pixels") in the binary image 
as either being part of a nucleus or not part of a nucleus. 
The result of the operation of the neural network is a digital 
map of the regions within the cytoplasm that are considered to 
be nuclear material. The nuclear material map is then 
subjected to further processing. The neural network 2 0 
according to the present invention is described in detail 
below. 

[0059] Following the application of the neural network 20, the 
resulting nuclear material map is subjected to an erosion 
operation (block 22) . The erosion operation 22 eliminates 
regions of the nuclear material map that are too small to be 
of diagnostic significance. The result is a modified binary 
map of nuclear regions. 
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[0060] The modified binary map resulting from the erosion 
operation 22 is then subjected to a dilation operation (block 
24) . The dilation operation 24 is subject to a no- join 
condition, such that, the dilation operation does not allow 
two separate regions of nuclear material to join to make a 
single region. In this way the accuracy of the binary map is 
preserved notwithstanding the dilation operation. The dilation 
operation 24 is preferably applied twice to ensure a proper 
filling of voids. The result of these dilation operations is a 
modified binary map of nuclear material. 

[0061] Following the dilation operation 24, an erosion 
operation is applied (block 26) . Double application of the 
erosion operation 26 eliminates regions of the nuclear 
material in the binary map that are too small to be of 
diagnostic significance. The result is a modified binary map 
of nuclear regions . 

[0062] The remaining operations involve constructing a binary 
map comprising high gradients, i.e. boundaries, of pixel 
intensity, in order to sever nuclear regions that share high 
gradient boundaries . The presence of these high gradient 
boundaries is evidence of two, closely spaced but separate 
nuclei . 

[0063] The first step in severing the high-gradient boundaries 
in the nuclear map is to construct a binary map of these high 
gradient boundaries using a threshold operation (block 28) 
applied to a Sobel map. 

[0064] The Sobel map is generated by applying the Sobel 
gradient operator to the 577 nm 8 -bit digitized image to 
determine regions of that image that contain high gradients of 
pixel intensity (block 29) . (The 8-bit digitized image for the 
577 nm band was obtained from the leveling operation in block 
12.) The result of the Sobel operation in block 29 is an 8- 
bit map of gradient intensity. 

[0065] Following the threshold Sobel operation 28, a logical 
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NOT operation is performed (block 30) . The logical NOT 
operation 30 determines the coincidence of the two states, 
high-gradients and nuclei, and reverses the pixel value of the 
nuclear map at the point of the coincidence in order to 
eliminate it from regions that are presumed to be nuclear 
material. The result of this logical operation is a modified 
nuclear map. 

[0066] The modified nuclear map is next subjected to an 
erosion operation (block 32) . The erosion operation 32 
eliminates regions in the modified nuclear map that are too 
small to be of diagnostic significance. The result is a 
modified binary map of nuclear regions. 

[0067] After the application of the gradient technique for 
severing close nuclear boundaries (blocks 28 and 30) and the 
erosion operation (block 32) for. clearing the image of 
insignificant regions, the binary map of nuclear regions is 
dramatically altered. To restore the map to its original 
boundaries while preserving the newly- formed separations, the 
process applies a dilation operation at block 34. The dilation 
operation 34 includes the condition that no two nuclear 
regions will become joined as they dilate and that no nuclear 
region will be allowed to grow outside its old boundary as 
defined by the binary map that existed before the Sobel 
procedure was applied. The dilation operation 34 is preferably 
applied four times. The result is a modified binary map of 
nuclear material . 

[0068] With the application of the dilation operation 34, the 
nuclear segmentation procedure according to the multi-spectral 
segmentation process 1 is complete and the resulting binary 
nuclear map is labeled in block 36, and if required further 
image processing is applied. 

[0069] As described above, the operation at block 20 in Fig. 1 
comprises neural network processing of the digitized images. 
In general, the neural network 2 0 is a highly parallel, 
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distributed, information processing system that has the 
topology of a directed graph. The network comprises a set of 
"nodes" and series of "connections" between the nodes. The 
nodes comprise processing elements and the connections between 
the nodes represent the transfer of information from one node 
to another. 

[0070] Reference is made to Fig. 2 which shows a node or 
processing element 100a for a backpropagat ion neural network 
20. Each of the nodes 100a accepts one or more inputs 102 
shown individually as a^, a^ , a3 . . . an in Fig 2. The inputs 
102 are taken into the node 100a and each input 102 is 
multiplied by its own mathematical weighting factor before 
being summed together with the threshold factor for the 
processing element 100a. The processing element 100a then 
generates a single output 104 (i.e. bj) according to the 
"transfer function" being used in the network 20. The output 
104 is then available as an input to other nodes or processing 
elements, for example processing elements 100b, 100c, lOOd, 
lOOe and lOOf as depicted in Fig. 1. 

[0071] The transfer function may be any suitable mathematical 
function but it is usual to employ a "sigmoid" function. The 
relationship between the inputs 102 into the node 100 and the 
output 104 is given by expression (1) as follows: 

b = { S Wji a^ - 9i } ' (1) 

[0072] where bj is the output 104 of the node 100, ai is the 
value of the input 102 to the node labeled "I", Wji is the 
weighting given to that input 102, and Gj is the threshold 
value for the node 100. In the present application, the 
transfer function is modeled after a sigmoid function. 
[0073] In its general form, the nodes or processing elements 
for the neural network are arranged in a series of layers 
denoted by 106, 108 and 110 as shown in Fig. 3. The first 
layer 106 comprises nodes or processing elements 112 shown 
individually as 112a, 112b, 112c, 112d and 112e. The first 
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layer 106 is an input layer and accepts the information 
required for a decision. 

[0074] The second layer 108 in the neural network 20 is known 
as the hidden layer and comprises processing elements 114 
shown individually as 114a, 114b, 114c, 114d and 114e. All of 
the nodes 112 in the input layer 106 are connected to all of 
the nodes 114 in the hidden layer 108. It will be understood 
that there may be more than one hidden layer, with each node 
in the successive layer connected to each node of the previous 
layer. For convenience only one hidden layer 108 is shown in 
Fig. 3. 

[0075] The (last) hidden layer 108 leads to the output layer 
110, The output layer 110 comprises processing elements 116 
shown individually as 116a, 116b, 116c, 116d and 116e in Fig. 
3. Each node 114 of the (last) hidden layer 108 (Fig. 3) is 
connected to each node 116 of the output layer 110. The output 
layer 110 renders the decision to be interpreted by subsequent 
computing machinery. 

[0076] The strength of the neural network architecture is its 
ability to generalize based on previous training of particular 
examples. In order to take advantage of this, the neural 
network is presented a series of examples of the type of 
objects that it is destined to classify. The backpropagation 
neural network organizes itself by altering the multiplicity 
of its connection weights and thresholds according to its 
success in rendering a correct decision. This is called 
supervised learning wherein the operator provides the network 
with the information regarding its success in classification. 
The network relies on a standard general rule for modifying 
its connection weights and thresholds based on the success of 
its performance, i.e. back-propagation. 

[0077] In the context of the multi-spectral segmentation 
process, the multi-spectral images are divided into two 
classes : 
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[0078] Co- cytoplasm and Ci - nuclear, separated by the multi- 
dimensional threshold t which comprises a 3 -dimensional space. 
The distribution of the pixels for the nuclear and cytoplasm 
objects is complex and the 3-D space comprises numerous 
clusters and non- over lapped regions. It has been found that 
the optimal threshold has a complex non- linear surface in the 
3-D space, and the neural network according to the present 
invention provides the means for finding the complex threshold 
surface in the 3-D space in order to segment the nuclear and 
cytoplasmic ob j ects . 

[0079] According to this aspect of the invention, the neural 
network 20 comprises an input layer 106, a single hidden layer 
108, and an output layer 110. The input layer 106 comprises 
three nodes or processing elements 112 (Fig. 3) for each of 
the three 8 -bit digitized values for the particular pixel 
being examined. (The three digitized values arise from the 
three leveled images collected in each of the three optical 
bands , as described above with reference to Fig . 1 . ) The 
output layer 110 comprises a single processing element 116 
(Fig. 3) which indicates whether the pixel under examination 
is or is not part of the nucleus . 

[0080] Before the neural network 20 can be successfully 
operated for decision-making it must first be "trained" in 
order to establish the proper combination of weights and 
thresholds. The training is performed outside of the 
segmentation procedure on a large set of examples. Errors made 
in the classification of pixels in the examples are 
"back-propagated" as corrections to the connection weights and 
the threshold values in each of the processing units. Once the 
classification error is acceptable the network is "frozen" at 
these weight and threshold values and it is integrated as a 
simple algebraic operation into the segmentation procedure as 
shown at block 2 0 in Fig 1. 

[0081] In a preferred embodiment, the neural network 20 
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according to the invention comprises a probability Projection 
Neural Network which will also be referred to as a PPNN. The 
PPNN according to the present invention features fast training 
for a large volume of data, processing of mult i -modal non- 
Gaussian data distribution, good generalization simultaneously 
with high sensitivity to small clusters of patterns 
representing the useful subclasses of cells. In another 
aspect, the PPNN is well -suited to a hardware -encoded 
implementation . 

[0082] The PPNN according to the invention utilizes a 
Probability Density Function (PDF) estimator. As a result, the 
PPNN is suitable for use as a Probability Density Function 
estimator or as a general classifier in pattern recognition. 
The PPNN uses the training data to create an N-dimensional PDF 
array which in turn is used to estimate the likelihood of a 
feature vector being within the given classes as will now be 
described. 

[0083] To create and train the PPN network, the input space is 
partitioned into m x m yi , . , m discrete nodes (if the 
discrete input space is known, then in is usually selected less 
than the range) . For example, for a 3-D PDF array creating a 
2^ X 2^ X 2^ grid is sufficient. 

[0084] As shown in Fig. 4, the next step involves mapping or 
projecting the influence of the each training pattern to the 
neighbor nodes. This is accomplished according to expression 
(2) as shown below: 

Pj[Xo/Xl Xn-l] = Pj-l[Xo/Xl Xn-l]+ Cij [Xo/Xl Xn-ll • 

1, if r^ - 0 

0. if rk > r^ (2) 

<^3 [XoiXi Xn-i] = { 1-rx , 

if rk < ro 

2"" -1 

E (1 - ri) 
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[0068] where Pj[Xo/Xi Xn-il is the current value of the 

[Xo/Xi Xn-i] node after the j ' th iteration; 

dj[Xo/Xi Xn-i] represents the influence of j • th input 

pattern to the [Xo/Xi Xn-il node; rk is the distance from 

the pattern to the k*th node; r^ is the minimum distance 
between two neighbor nodes; and n is the dimension of the 
space . 

[0069] From expression (1), it will be appreciated that 

Vj Z dk{j)-l 

represents the normalized values. 

;c=i 

[0070] Once the accumulation of Pn[Xo/Xi Xn-il (where j = N 

number of the training patterns) is completed, a 
normalization operation is performed to obtain the total 
energy value for PPNN SppN - 1. The normalized values (i.e. P*) 
for PPNN are calculated according to expression (3) as 
follows : 

P*n[Xo/Xi Xn-i] = Pn[Xo/Xi Xn-i] /N (3) 

[0071] For feed-forward calculations the trained and 

normalized nodes P*n[Xo,Xi Xn-il and the reverse mapping are 

utilized according to expression (4) given below, 

2''-l 

hjLXo,. . Xn-l3 - S P*^' [Xo,Xl,. . . Xn-l] dj^^* [Xo/ Xl Xn-ll , (4) 

N 

i = 0 

where dj*^*[Xo/Xi Xn-i] are calculated according to 

expression (1) above. 

[0072] To solve a two class (i.e. Co - cytoplasm and - 
nuclear) application using the PPNN according to the present 
invention, two networks must be trained for each class 

separately, that is, PcoLXo/Xi Xn-il and Pci[Xo/Xi Xn-il - 

Because both PPNN are normalized, they can be joined together 
according to expression (5) below as follows: 
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Pco/ci[Xo/Xi Xn-il and P*co[Xo/Xi Xn-il - P*ClCXo/Xl Xn-J (5) 

The final decision from expressions (4) and (5) is given by 

Cq, if hj > 0 

Patternj e Ci, i£ hj < 0 (6) 

[0073] While the PPNN according to the present invention is 
particularly suited to handle multi-modal data distributions, 
in many practical situations there will be an unbalanced data 
set. This means that some clusters will contain less data 
samples than other clusters and as a result some natural 
clusters which were represented with a small number of 
patterns could be lost after PPNN joining.. To solve this 
problem there is provided an algorithm which equalizes all 
natural clusters according to another aspect of the invention. 

[0074] Reference is next made to Fig. 5, which shows in flow 
chart form an embodiment of a clustering algorithm 200 
according to the present invention. All training patterns, 
i.e. N samples, in block 202 and a given number (i.e. "K") of 
clusters in block 204 are applied to a K-mean clustering 
operation block 206. The clustering operation 206 clusters 
the input data and generates clusters 1 through K (block 208) . 
Next, all the training data which belongs to an i*^^ -cluster is 
extracted into a separate subclass. For each sub-class of 
training data, a normalized PPNN, i.e. Ei = 1, is created 

(block 210) . The final operation in the clustering algorithm 
comprises joining all of the K PPNN's together and normalizing 
the resulting PPNN by dividing all nodes by the number of 
clusters (block 212) . The operation performed in block 212 may 
be expressed as follows: 

E = (El + .... + Ejc) /K-1 

[0075] It will also be understood that the clustering 
algorithm 200 may be implemented to the each class separately 
before creating the final classifier according the expression 

(6) above, as follows. The optimal number of clusters for 
each of two classes may be found from final PPNN performance 
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analysis (expression (6) above) . First, the number of 
clusters for PPN^ = 1 are fixed and the optimal number of 
clusters for PPN^ are found. Next, the reverse variant is 
modeled as: PPN^ = 1/ a PPN^ = opt. Lastly, the two optimal 
networks PPNi°^^ a PPN^^pt are combined together according to 
expression (6) . 

[0076] While the neural network assisted multi-spectral 
segmentation process is described with a Probability 
Projection Neural Network according to the present invention, 
it will be understood that other conventional neural networks 
are suitable, including for example, Backpropagation (BP) 
networks, Elliptic Basic Functions (EBF) networks, and 
Learning Vector Quantization (LQV) networks. However, the 
PPNN is preferred. The performance results of the Probability 
Projection Neural Net have been found to exceed those achieved 
by conventional networks, 

[0077] According to another aspect of the present invention, 
the neural network assisted multi -spectral segmentation 
process is implemented as a hardware -encoded procedure 
embedded in conventional FPGA (Field Programmable Gate Array) 
logic as part of a special-purpose computer. 

[0078] The hardware implementation of this network is found in 
the form of a look-up table contained in a portion of hardware 
memory (Fig. 6) . As described above, the neural network 2 0 
comprises three input nodes and a single, binary output node. 
The structure of the neural network 2 0 according to the 
present invention also simplifies the hardware implementation 
of the network. 

[0079] As shown in Fig. 6, the three input nodes correspond to 
three optical bands 301, 302, 303 used in gathering the 
images. The images taken in the 530 nm and 630 nm bands have 

7- bits of useful resolution while the 577 nm band retains all 

8- bits. (The 577 nm band is centered on the nucleus.) The 
performance of the neural network 2 0 is then determined for 
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all possible combinations of these three inputs. Since there 
are 22 bits in total, there are 2^^ or 4.2 million possible 
combinations. To create the look-up table, all input pixels in 
the space (2*^ x 2*^ x 2^ variants for the three images in the 
present embodiment) are scanned and the look-up table is 
filled with the PPNN decision, i.e. 1 - pixel belongs to 
nuclear; 0 - pixel doesn't belong to nuclear, for all each of 
these pixel combinations. 

[0080] The coding of the results (i.e. outputs) of the neural 
network comprises assigning each possible combination of 
inputs a unique address 3 04 in a look-up table 3 05 stored in 
memory. The address 3 04 in the table 3 05 is formed by joining 
together the binary values of the three channel values 
indicated by 306, 307, 308, respectively in Fig. 6. For 
example, as shown in Fig. 6, the pixel for the image from the 
first channel 301 (i.e. 530 nm) is binary 0101011, the pixel 
for the image from the second channel 302 (i.e. 630 nm) is 
binary 0101011, and the pixel for the image from the third 
channel 3 03 (i.e. 577 nm) is binary 00101011, and concatenated 
together binary representations 306, 307, 308 form the address 
304 which is binary (0101011010101100101011) . The address 304 
points to a location in the look-up table 3 05 (i.e. memory) 
which stores a single binary value 3 09 that represents the 
response of the neural network to this combination of inputs, 
e.g. the logic 0 at memory location OlOlOllOlOlOllOOlOlOll 
signifies that the pixel in question does not belong to the 
nucleus . 

[00 81] The hardware -encoding of NNA-MSS advantageously allows 
the process to execute at a high speed while making a complex 
decision. Secondly, as experimental data is further tabulated 
and evaluated more complex decision spaces can be utilized to 
improve segmentation accuracy. Thus, an algorithm according 
to the present invention can be optimized further by the 
adjustment of a table of coefficients that describe the 
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neural -network connection weights without the necessity of 
altering the system architecture. 

[0082] The present invention may be embodied in other specific 
forms without departing from the spirit or essential 
characteristics thereof. Therefore, the presently discussed 
embodiments are considered to be illustrative and not 
restrictive, the scope of the invention being indicated by the 
appended claims rather than the foregoing description, and all 
changes which come within the meaning and range of equivalency 
of the claims are therefore intended to be embraced therein. 




